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Abstract 
Producing quality educational subjects has become a major 
concern of many institutions. Driven by the need to compete in 
a more global market and with government budgets shrinking, 
many universities are turning to the adoption of new learning 
technologies to expand their markets and improve the 
flexibility of their offerings. This paper considers how the 
economic realities associated with the new market place can 
be reconciled and even contribute to the development of 
quality educational products and outcomes. In this paper we 
explore a model of organisational behaviour based on a series 
of interlocking quality improvement cycles. We use this to 
examine current practice at RMIT and identify areas to focus 
on for improving performance and effectiveness of a range of 
institutional initiatives, associated with an institutional 
approach to coordinating the adoption of new technologies for 
online flexible learning and student management. 
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Introduction 


The nature of online learning makes more public than ever the question of 
quality. The website of any organization is becoming the modern shop- 


front; so institutions have to understand the nature of this new medium in 
order to get the most value from it in a marketing sense. The challenge is 
to use the technology to produce quality educational outcomes. This raises 
two questions: 
e What is a quality online educational experience? 
¢ How is the development of quality educational experiences 
measured and encouraged? 


At RMIT, we are currently wrestling with both these questions. Producing 
quality educational experiences using online technology is a complex and 
costly activity. The development and implementation of new learning 
technologies can prove to be an economic black-hole unless there is 
careful planning and evaluation of the implications occurring at all levels 
of the institution. What happens in teaching has always been influenced by 
the context in which the classroom is situated. The production of quality 
online educational experiences involves consideration of many issues: 
educational design, interface design, copyright and intellectual property, 
content development, staff training, student induction, technical 
infrastructure, establishment of support mechanisms and the 
rationalization of administrative procedures. Many of these issues are out 
of the control or expertise of the teacher, and so will require a new way of 
working. 


The New Learning Environment 


An institution may have a range of motivations for adopting new learning 
technologies (NLT). NLT may be seen as a means of: maintaining its 
position by providing flexible courseware more responsive to student 
needs; introducing cost savings and efficiencies; and/or rationalising 
business processes and automating administrative tasks. Whatever the 
motivation, the implementation of new learning systems on an institutional 
level is complex. From an institutional perspective in the current economic 
climate, there is an imperative to get a return on the investment (ROI). 
Ultimately, though, a university will be judged by the quality of the 
teaching and learning it offers its students. To improve the learning 
experiences offered, it is essential that there is an eye for improvement 
from the level of the individual subject teacher up through the organisation 
to the activities of senior managers. Inglis, Ling and Joosten (1999, p. 143) 
comment: 


This does not imply that the vision of the project and the planning of its 
implementation need to be a top-down process. On the contrary, there needs to be 


ownership, vision and enthusiasm at all levels of the organisation. 
Institutional quality improvement processes 


There has been a concerted effort in recent times to set up quality 
processes within many institutions. This has led to the development of 
student feedback systems which individual teachers and course teams may 
use to improve the next offering of the subject. Such a system is 
predicated upon the evaluation of an educational activity or project leading 
to improvements in subsequent attempts; this is congruent with the action 
inquiry process outlined in Phillips, McNaught, Rice and Tripp (2000). 
The learning from this process is too precious to be left untapped within 
any one subject or minor project. Unless the information is shared, the 
wider institution does not necessarily benefit as much from these projects. 
Much can be gained by facilitating the sharing of the new knowledge and 
experience across the institution. Laurillard (1997) contends: 


The university must have a technical and pedagogical innovative environment for 
R&D projects providing opportunities to trial and experiment and to collect 
feedback on these via the quality assurance process. ... Many such pilot 
experiments in universities have been conducted in isolation from the universities 
management process. 


Unless the evaluation occurs in the context of the institutional processes at 
large, then the valuable learning opportunities inherent in these projects 
will be lost to the institution. 
Bain (1999, p. 170) considered evaluation reports of several projects, and 
concluded that the context in which an innovation is occurring has to be 
considered. 

...the benefits were short-lived and/or did not transfer. This finding offers a 


salutary caution to all educational innovators and underscores the need to view 
innovation within the institutional contexts in which it will thrive or die 


Thus the institutional quality processes need to be such that the culture and 
procedures encourage the flow of information across subject, course, 
departmental and faculty boundaries. It is the contention of this paper, that 
where quality cycles do not enable this flow of information, the lessons 
learned do not easily go beyond the subject concerned, students do not 
benefit and the ROI is reduced 


A model for promoting institutional quality processes 


Laurillard, (1993) proposed the ‘conversational model’ of learning where 
a ‘conversation’ can be considered as a two-way flow of information. In 
essence, she contends that learning occurs when the student acts for a 


particular purpose and then receives feedback on that action. The student 
then assimilates and reflects upon the feedback in order to re- 
conceptualise and articulate a new understanding to the teacher. This is a 
classical action research cycle of goal-action-feedback-modified action 
integral to quality improvement processes. The critical part of the process 
is the reflection. Biggs (1999, p. 60) supports the importance of reflection 
as part of the learning cycle: “my own assumption is that helping teachers 
to improve their teaching is best done using a theory that helps teachers 
reflect on what they are doing”. Laurillard (1997) claims that a similar 
model of learning can be applied at the institutional level. 

The educational institution has to be able to be responsive to change 

... able to reflect on the degree of convergence that is achieved in the 

learning conversation and construct an adapted environment. ... Its 

internal structure ought to be similar to the conversational 

framework for an individual if it is to learn from experience. 


We can then consider the meaning of ‘learning’ as far as an institution is 
concerned. It means getting better at what it does, learning from 
experience. There is no doubt that the adoption of new learning 
technologies is, for most institutions and staff within them, a learning 
experience. Feedback obtained on a project or program is used to inform 
the next iteration or to inform new or related projects. 


Establishing an institutional learning conversation 


As indicated earlier, it is not enough to have ‘learning conversations’ or 
‘quality improvement cycles’ operating at distinct levels within an 
organisation. There must be overlap so that these conversations occur 
across boundaries. For institutional learning to the project team must have 
a conversation with the institution. In the context of a project, to improve 
the teaching and learning in a subject, the teacher becomes not only a 
researcher of a discipline, but a researcher in how to teach the discipline, 
which is the fourth Boyer Scholarship, Boyer (1990). A similar point was 
also made by made by Laurillard (1993). The benefits of the project team 
sharing its learning with other staff will lead to improved learning 
outcomes for a wider range of students and staff. With the restricted 
budgets at play in the economic environment in which today’s institutions 
operate, it is too costly for projects to be funded without any institutional 
benefit coming from them. 


In fact, one can imagine an 
almost unlimited series of 
overlapping learning 
conversations (see Figure 1) 


[ going on within an organization 
driven by feedback and reporting 
eal procedures. It is in this way that 
a culture of quality improvement 
[ has to pervade an organization in 
| order to get the best results. 
Quality is achieved over time. 
[ An institution may have an ideal 
which it strives to reach, but it 
must take the view that 
numerous and continuous cycles 
are required to move towards the 
ideal. The ideal itself may never 


Figure 1: Overlapping quality be achieved since it may alter as 
improvement cycles needs and technology change. 


Quality Subject and Courseware Production 


Quality assurance criteria should be seen as a means of gauging what has 
been achieved and what needs to be improved. Quality cannot be 
mandated. When we consider the complexity of introducing new learning 
technologies, an iterative approach to improving quality is very important. 
Taylor (2000) explains this in terms of nurturing cultural change. To 
promote quality in teaching and learning is a complex task. In the 
classroom, it has to do with the teaching practices, student capabilities, 
resources, design of the subject and the types of assessment undertaken. 
Many staff may need professional development in some of these areas to 
go along with the training and professional development associated with 
the use of new learning technologies themselves. Taylor (2000) states: 
Some faculties will have to go through what might look like a sub- 
optimal phase of development, but only by doing so will they be 
able to move forward. This is particularly relevant in an environment 
where implicit teaching knowledge and expertise will need to be 
articulated before a multimedia approach to the subject can be 
considered or designed. 


At RMIT, the strategy is to use the new technology as a means of 
encouraging a re-thinking of teaching and learning practices, not simply 
transferring established practice to an online environment. In this case, the 
technology becomes a medium to promote professional growth. Teachers 
may therefore feel uneasy on two accounts; they are learning new skills 
while operating within an unfamiliar environment. There is also a range of 
other considerations necessary to produce quality courseware in the online 
environment: copyright and intellectual property issues; design and 
usability issues; navigation issues; accessibility issues. Without 
institutional support, many teachers will feel such a task is overwhelming. 
It is clear that institutional support, professional development for staff and 
a staged process of development will be the only realistic ways to develop 
quality subjects. It is the function of management to set the scene for this 
renewal to occur by developing policies and providing the resources and 
support to teaching staff. The outcomes of the educational programs can 
then be monitored to evaluate the effectiveness of these policies. 


The RMIT Experience 


In 1998, RMIT commissioned a team to investigate how the information 
technology policies and the Teaching and Learning Strategy of the 
University could be aligned. This group produced a report “The Education, 
Training and Information Technology Alignment Report’ which became 
known as the ITAP Report. A team was set up to manage the development 
and implementation of an institutional system using IT to enhance and 
support the administrative and educational functions of the University. 
The ITAP Team, now called Learning Technology Services (LTS), 
developed the ‘RMIT Distributed Learning System’ (DLS) as a web-based 
set of tools to support teaching and learning activities. 


Prior to the DLS, there was a lot of activity across RMIT to produce 
online learning experiences. Individual staff had developed and used web- 
based learning materials; on diverse software platforms. The DLS was the 
first attempt at an institutional wide learning system. The benefits of this 
approach to the institution and the individuals were obvious: providing 
central support, training, greater security, better linking with 
administrative systems, quality assurance processes, a consistent interface 
etc. Initially, it was unclear how this corporate initiative would be viewed 
by the RMIT community, but it soon became apparent that there was a 


large number staff waiting to begin the process of putting learning 
material online. 


Version of the DLS Number of registered subjects 


Benchmark-semester one 1999 


Semester two 1999 225 


Semester one 2000 683 


Table 1: Growth of number of subjects registered in the RMIT DLS 


The experience of the Benchmark in semester one 1999 (McNaught et al., 
1999) made it clear that a more systematic approach was needed. Each 
faculty was asked to develop “Course and Subject Renewal Plans’ in 
accordance with a set of guidelines produced by the ITAP team, and this 
plan was to be used to identify subjects for renewal according to faculty 
priorities. This was seen as a means of directing the resources 
strategically. Along with this, the DLS developed a DLS Planning Tool. 
This tool was produced in May 1999 and was released two months before 
the start of semester two 1999. It was to assist staff to identify consider the 
many issues which require consideration, when developing a subject for 
online delivery. Such issues included: 

clearly defining the educational purpose, 

the content development support required, 

the level of interactivity being considered, 

the expertise available to the development team, 

the professional development and training required by staff, 

the student induction and access issues, 


the intellectual property issues, and 
resources available. 


The Planning Tool was to be used by key faculty personnel to work with 
the leaders of the subjects. Only about six of the subject coordinators (out 
of 225) actually returned a completed form. Many staff did not even hear 
about the form and there were some complaints about the level of detail it 
required. After one such complaint, the form was modified to a one page 
application form (compulsory) and a detailed planning guide (optional) for 
semester one 2000. There is no evidence of any subject team making use 
of the optional planning guide. Following an evaluation report of the DLS 
in semester two 1999, (Kenny, 2000), the issue of quality and planning to 
go online has become a major focus for the DLS during 2000. In May 
2000, all subjects using the DLS were reviewed against university quality 
assurance guidelines and evaluative feedback was provided on all subjects. 


Drivers for Institutional Change at RMIT 


There has been substantial investment by RMIT to promote quality 
flexible learning outcomes. The investment is quite considerable, with 
approximately 5% of each faculty budget being set aside, along with 
central money, to fund the course and subject renewal process. Also, an on 
going major upgrading of the RMIT network, and student and staff 
computer facilities has occurred over the last two years. A list of the major 
institutional drivers is given in Table 2 with some commentary on the 
effects of each. Many of these drivers are yet to bear fruit. Our contention 
is that although there has been considerable effort to get institutional 
involvement in the initiative, because the underlying quality improvement 
feedback and reporting loops are not yet in place, much of the strategy is 
seen to be top-down, with insufficient information reaching the staff or 
students. Figure 2 shows a proposed RMIT operating model. The heavy 
boxes represent the boundaries of the organization across which 
information must flow in both directions. The large black arrows represent 
information flowing across these boundaries. Many more boundaries 
could be added (e.g. departments, support services, etc.) but have not been 
added for simplicity. While this model is set-up for the context of RMIT 
University, it is also generic enough to be adapted to other institutions. 
The arrows in Figure 2 which are dotted represent the information flows 
which are either not in operation as yet, or which operate on an ad hoc or 
isolated basis. The diagram shows several gaps in the current institutional 
processes at RMIT and future attention needs to be devoted to establishing 
these communication flows and promoting the culture in which they will 
operate. 


Planning and project management 


The Course and Subject Renewal Strategic Planning Process was 
developed in mid-1999. The process had its problems as any new process 
will: it was not clearly understood by development teams, the 
development timelines were too short, the resources needed were not 
clearly identified, and there was no mechanism put in place to adapt and 
improve the process despite the two evaluation reports (McNaught et al., 
1999; Kenny, 2000). For quality improvement to occur, this evaluation 
information must be used to improve operations. 


| Institutional Driver Comment | 


The RMIT Teaching and Learning | This has been a key strategy for driving change in the 
The ITAP report had 113 | The ITAP report is driving much of the change 
recommendations about linking | associated with the DLS and the Academic 

IT developments with teaching | Management System (AMS, see below) 

and learning needs. 

The institutional target of 60% of | This target has generated a lot of interest and activity. 
all subjects with some flexible Led to rapid growth of number of subjects within the 
delivery components by the end of | DLS. The target itself takes no account of the iterative 
2000. development required to produce quality outcomes. 
Creation of Directors of | Key people in the adoption and promotion of the 
Teaching Quality (DoTQs) and | institutional strategy. Important in developing 
Directors of Information | workable quality assurance loops across the institution 
Technology (DoITs) posts in | and shaping them to work within the context of their 
each faculty as resources to | own faculties. 

direct and influence policies. 

Development of an Academic Major initiative to develop a computerised system to 
Management System (AMS) for streamline and standardise many of the university 
mid-2001. administrative procedures. 


A Business Process Re- 1999 project which reviewed the administrative 


engineering (BPR) review of the processes of the university in preparation for the 
university administrative systems _| implementation of the AMS. 
Establishment of the Distributed DLS operates using a secured central server system on 
Learning System (DLS) the RMIT network. Support, professional development 
and training offered to RMIT staff and students. 
Faculties asked to develop Have the potential to promote meaningful change, but 
e IT Strategy Plans for the | little evaluation and reporting occurs. Feedback and 
strategic re-development | accountability loops do not function well, so there is 
of IT infrastructure. little direct evidence of the effectiveness of each 
Course and Subject strategy and how it might be improved. It is hard to get 
renewal guidelines for an institutional picture, or it takes too long for 
the strategic re- problems to be identified. 
development of subjects 
and courses. 
Selection and _ training of | 145 staff trained to work with staff in their own 
Learning Technology Mentors | departments, to facilitate communication between the 
(LTMs) within each department | staff and the DLS and to mentor them as they renew 
to work with staff. (See | their subjects The process is not necessarily supported 
McNaught & Kennedy, 2000). by all faculty and departmental processes. 


New QA processes for online | These have just been formally accepted by the 

subjects University after extensive consultation. 

Staff workplans Staff are required to produce a workplan to set out 
expectations linked with their professional 


development for the year. Need to more closely link 
the workplans with involvement in strategic renewal 
projects and to promote evaluation and reflection. 


Table 2: Institutional drivers for change within RMIT 


Faculty Level 


approves 
resources 


T 
Project planning and development cycle 


Course/Subject 
Team Level 


Project Report to Faculty 
@ outcomes and plans for next iteration 


Faculty Report on its renewal strategy: Aggregate reports from 
all its renewal project activities. 


e Faculty Priorities, Plans and ROI 


Institutional Level — 
Institution Report: Aggregate faculty reports and LTS 


Institution Priorities and ROI 


Renewa 
1 


Figure 2 RMIT model for institutional quality processes for course 
renewal 


Research indicates that a successful project should appoint a project 
manager to oversee its development and implementation (e.g. Alexander, 
McKenzie and Geissinger, 1998). In the model of figure 2, it is proposed 
that subjects or courses for development are identified strategically by the 
faculty processes. It is also wise to institute a feasibility process to explore 
the scope and likely benefits and costs associated with any significant 
project proposal before it is begun. Each becomes a development project 
in its own right and is resourced accordingly and progresses according to a 
recommended project planning process. The subject goes into its delivery 
phase during which evaluation data is collected which forms the basis of 
the project reporting. 

Evaluation and reflection are critical to articulating the learning and 
improving quality. Therefore the processes need to enable this information 
to pass between the various groups. The reporting procedures are a means 
of achieving this information flow. For example, a subject team receives 
feedback on the operation of the subject which needs to be analysed and 
reported on to improve the subject for the next iteration. Some of this 
feedback information is also be required by the faculty (or department). 
Not all the information may be relevant, but a means for the faculty to 
capture the relevant information and aggregate it with that from other 
projects, analyse and report on it, will enable the faculty to gauge the 
effectiveness of its Course and Subject Renewal Plan and influence the 
next iteration of the Plan. Similarly, the institution as a whole needs some 
of the information so that an evaluation of the institutional strategic 
approach can be made. Hence the reports of each faculty need to be 
aggregated. 


Culturally, it involves quite a degree of change for most staff to adopt this 
approach to development. Depending on the complexity of each project, it 
most likely will require a development team with a range of expertise. 
While there are examples of individual teachers and subject teams 
producing excellent work, most academic and teaching staff at RMIT are 
not used to working so closely with non-teaching staff such as graphic 
designers, IT support staff and so on. The fact that staff in the process will 
need to explicitly reflect on and evaluate their teaching practice will lead 
to better quality teaching. The reporting mechanisms are important so that 
the documentation of the learning occurs and that it is shared with 
colleagues. 


Quality Assurance and Systems Supporting Online Teaching 


A quality improvement process relies on having evaluation information on 
which to base decisions for the next cycle. In many cases, the timelines 
and demands set out for the DLS project were far too tight, so that 
planning and specification for the next version was often done before the 
previous version was properly evaluated. For instance, at the system 
infrastructure level, there was insufficient time allocated to properly 
develop and test the delivery systems. Delays in gaining approval for 
purchasing servers further reduced the time available, which led to other 
delays in implementing the processes to set up and test the software 
developments. The time scales are critical in initiatives such as these. In 
both semesters, the DLS operational systems were not released for staff 
use until after the semester had begun. The subsequent bugs and technical 
problems which occurred should have been discovered in a testing phase 
rather than a live delivery phase. Much of this information is contained in 
an evaluation report, Kenny (2000). 


The consequences of the lack of coordinated planning has been shown up 
during a recent quality review of the subjects registered in the DLS. As 
mentioned earlier, there was an institutional target set to achieve “60% of 
subjects with a flexible delivery component” by the year 2000. This 
became misinterpreted across RMIT as meaning “60% of subjects with an 
online component’. Experience has shown that this target does not 
account for the complex reality of producing quality learning systems and 
subjects. It seems that goals set too high can be just as costly as those set 
too low. 


The message is to establish clear lines of responsibility with planning and 
evaluation and reporting cycles in place. The managing body at each level 
can then report on progress to the wider RMIT community and thereby 
influence policy development. In order for problems to be minimised in 
future, there has to be consultation on the development of a process, which 
when adopted, has to be backed up with adequate resourcing and high 
quality support resources and professional development activities. Of 
course the process itself will be improved and refined during subsequent 
quality improvement cycles also. 
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